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ABSTRACT 


The article aims to outline and analyze the most demanding parts of computer-based exams for the students of the Ural 
Federal University named after the first President of Russian B.N. Yeltsin (Ekaterinburg, Russia) during one exam 
session of spring of 2016-2017. For this research, a quantitative method of mean scores for reading, listening, and use of 
English (for levels A2, B1, B2 CEFR) was used. The data expressed in mean percentages were processed and several 
conclusions were drawn, trying to account for the results. The results revealed that at the A2 level there are more 
difficulties with the Reading part (Reading and identifying appropriate vocabulary) for the first-year students than for the 
second-year students, 46% and 54% respectively. By contrast, in Listening (Gap-fill) students of the second year showed 
lower scores (50%) than those of the first one (67%). At B1 level, the greatest challenge was Reading part (Sentence 
transformation) for students of both years of education (38% for the first and 50% for the second year students). B2 level 
students demonstrated the most considerable number of issues, among them Use of English (Keyword transformation 
0 for both years of education, open cloze, word-formation), Reading (gapped text), Listening (Gap-fill, 
dialogue/monologue 6 multiple choice questions). 
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1. INTRODUCTION 


Since 2007 in accordance with the Bologna process higher education in Russia has been undergoing 
significant changes. One of the most important issues has been internationalization, in the broader sense ‘to 
enhance research and knowledge capacity and to increase cultural understanding’ (Altbach & Knight, 2007). 
All of them can be achieved only through better knowledge of English. Thus, in order to ensure graduates’ 
good command of this language, both the process of language learning and its subsequent assessment are of 
utmost importance. 

In the past couple of years, the students of the Ural Federal University named after the first President of 
Russia B.N. Yeltsin (Ekaterinburg) (hereinafter UrFU) have been learning the target language in the 
following way. First, students are divided into groups based on their level; Cambridge English placement test 
is used to determine the incoming level (using the Cambridge English scale of assessment). Having studied 
the language for 2 years, they must attain a higher level than the entrance one. During the first year of 
education, the test they take is of their incoming level, however, in their second year of education, the 
difficulty is increased and they are to take a higher level. 

To most productively assess students’ progress, UrFU has introduced Independent computer-based Test. 
This system (initially adopted in 2015) is a mock Cambridge English Language Assessment exam, including 
only receptive skills and use of English. The choice of a computer-based test system is justified since one of 
the major goals is the impartial assessment of the level of English-language proficiency. These tests, although 
costly, give instant feedback, reduce the amount of time the staff have to take to check the works, eliminate 
human-error possibility, ensure higher variability of the test (since the test pool is quite big, several students 
are unlikely to do the same task at the same time) (Rasskazova, Muzafarova, Daminova, & Okhotnikova, 
2017). 
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Our aim was to analyze the results of the Independent computer-based Test taken by Bachelor degree 
students of the first and second years of nonlinguistic faculties and departments of UrFU in order to 
determine the most challenging parts of the exam and observe whether there is any progress from the first 
year to the second one. 

Thus, the novelty of this study consists in the fact that most research dealing with computer-based tests 
investigate their outcomes as opposed to paper-based tests, aiming to determine whether the former put 
students under additional pressure. Our main aim, however, was to inform teachers of the most demanding 
parts of tests with a view to improving the results. 


2. RESULTS OF THE RESEARCH 


For the research, we chose the results of the exam taken at the end of the second term of the 2016/2017 
academic year. The levels and number of students can be seen in table 1. This sample is substantial for the 
analysis of general trends in success in test completion. 


Table 1. Number of test-taking students by level 


A2 (pre-intermediate) | B1 (intermediate) B2 (upper-intermediate) | Total 
Ist year 580 259 135 964 
2nd year 521 191 135 847 
Total 1101 450 270 1821 


All the figures here and later are expressed as mean percentage (out of 100%). Speaking about the 
average test results of students the most successful were the students of the second year at B1 level (71%) 
and with a minor difference at A2 level (69%). By contrast, the least successful were those at B2 level (62%) 
(66%) of the first and second year. These results might be predictable since the latter level requires the above 
average knowledge of the language, which consequently is the most demanding exam among all the others. 

Our next stage was to see what types of tasks from what part of the test were the most challenging. We 
compared the results obtained by the students of the same level but of different years. 


2.1 A2-Level Students’ Least Successfully Done Tasks 


For the students of the second year, the most challenging part was Listening (Gap-fill) 50% while for the first 
year students this task did not present substantial difficulties and was quite successfully done 67%. What was 
difficult for them was Reading with the task of reading and identifying appropriate vocabulary 46% as 
opposed to the second year’s 54%. 

Gap-fill task tests the ability to extract specific information from a dialogue or monologue. It has always 
been a problematic type of task for the students learning English as a second language. What is unexpected, 
though, is the fact that there was a lower rate of success among the second year students who must have had 
more time to get used to and prepare for the exam than those on their first year of education. 


2.2 B1-Level Students’ Least Successfully Done Tasks 


At this level, there are evident problems with the Reading part (Sentence transformations) 38% for the first 
year and 50% for the second. This task focuses on grammatical precision and requires students to complete 
five sentences, all sharing a common theme or topic. However, there is an expected improvement in the score 
demonstrated by the second year students. 

What was most surprising for us was the fact the second least successful task was Reading with the task 
of understanding of various kinds of short texts: authentic notices and signs, packaging information and 
communicative messages. The results of students of both years of education are the same 50%. A text is often 
accompanied by visual information related to its context. Students may either pay not enough attention to this 
type of task or have real difficulties in understanding the natural English speech from everyday life. We 
believe this must be further studied and thoroughly investigated. 
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Listening was challenging for the students of the first year 56% but slightly less than the Reading part. 
Despite this, the students of the second year did not experience the same problems with the Listening and had 
a higher average score of 67%. 


2.3 B2-Level Students’ Least Successfully Done Tasks 


At the given level, students demonstrated poor results in the Use of English part in the task that consists of 
six questions. Each question contains three parts: a lead-in sentence, a keyword, and a second sentence of 
which only the beginning and end are given. Students have to fill in the gap in the second sentence so that the 
completed sentence is similar in meaning to the lead-in sentence. The keyword must not be changed in any 
way. In this part of the paper, the focus is both lexical and grammatical and a range of structures is tested. 
Students of both years showed insufficient results with 0% success rate. 

Use of English (open cloze) presented difficulties as well. Students are required to draw on their 
knowledge of the structure of the language and understanding of the text in order to fill the gaps. In this part, 
as there are no sets of words from which to choose the answers, students have to think of a word which will 
fill the gap correctly. Students’ results here are 46% for the first year and 56% for the second. 

For the first-year students listening tasks appeared demanding as well as for the students of other levels. 
Interestingly at this level, not only is Gap-fill task problematic but also the extended monologue/dialogue was 
of the same difficulty 50%, whereas for the second year it was 73% and 67% respectively. This might be 
accounted for by the fact that the level presupposes the mastery of a much broader range of vocabulary and 
grammar, an additional challenge might be the length of the recording which is substantially longer than in 
the exams of the previous levels. 

Moreover, Reading part with the gapped text was similarly challenging for students of both first and 
second years 57% and 64%. This part is not present in the exams of the previous levels; consequently, the 
format itself might be foreign to students. 


3. CONCLUSION 


As a result of our study we determined that in order to improve the results of test-taking students, teachers 
should be aware of the following peculiarities: 

1) At A2 level, more attention should be paid to the Listening Gap-fill tasks, with special emphasis on the 
way to correctly transfer the answers which might account for these results; 

2) At BI level, writing sentence transformation, reading tasks (short discrete texts, signs and messages, 
postcards, notes, emails, labels) are particularly challenging. Like first-year students of the A2 level, 
first-year students of B1 level must improve their performance in Listening Gap-fill; 

3) At the B2 level, students tend to experience the greatest number of challenges. In addition, as the data 
suggest, the year of education only slightly influences the problematic parts of the exam. The main difference 
is that first-year students perform less successfully in Listening tasks (Gap-fill, multiple choice with a longer 
dialogue). For both years, Use of English tasks (Keyword transformation, Open cloze, Word formation) and 
Reading (gapped text) are difficult. 

Speaking about some universal challenges, Listening Gap-fill is especially tough. This may occur due to a 
number of reasons, for example, this task type requires short-term memory activation which does not receive 
the attention it deserves; students have trouble with authentic pronunciation; students do not know how to 
write the answers correctly. Thus, students should be taught useful strategies for the completion of these 
types of tasks; their vocabulary, as well as the knowledge of various accents should be expanded; lastly, work 
on phonetics is in order since recognizing chunks of words is invaluable for a more complete understanding 
of a text. 
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